Autonomous driving system with air support

ABSTRACT

Aspects an autonomous driving system with air support are described herein. The aspects may include an unmanned aerial vehicle (UAV) in the air and a land vehicle on the ground communicatively connected to the UAV. The UAV may include at least one UAV camera configured to collect first ground traffic information and a UAV communication module configured to transmit the collected first ground traffic information. The land vehicle may include one or more vehicle sensors configured to collect second ground traffic information surrounding the land vehicle, a land communication module configured to receive the first ground traffic information from the UAV, and a processor configured to combine the first ground traffic information and the second ground traffic information to generate a world model.

TECHNICAL FIELD

The present disclosure generally relates to the technical field ofautonomous driving, and specifically, relates to an apparatus and methodfor autonomous driving with air support.

BACKGROUND

Autonomous driving systems have been proposed to replace the manualdriving mode in which a vehicle travels under the control of the driver.An autonomous driving vehicle, or autonomous driving systems embeddedtherein, typically includes multiple sensors to detect the objectsaround the vehicle. Those objects should be promptly detected andlocated to avoid possible collision with the vehicle. Many of theexisting autonomous driving systems includes Light Detection and Ranging(LiDAR) devices, cameras, or Radio Detection and Ranging (radar)sensors.

However, none of these sensors can detect objects blocked by anotherobject, for examples, a pedestrian running behind another vehicle. Thosesensors also have difficulties to detect other vehicles in a lowvisibility weather. Even on a sunny day, the range of those sensors arealso limited to around one hundred meters, if not further.

SUMMARY

The following presents a simplified summary of one or more aspects toprovide a basic understanding of such aspects. This summary is not anextensive overview of all contemplated aspects and is intended toneither identify key or critical elements of all aspects nor delineatethe scope of any or all aspects. Its sole purpose is to present someconcepts of one or more aspects in a simplified form as a prelude to themore detailed description that is presented later.

One example aspect of the present disclosure provides an exampleautonomous driving system. The example autonomous driving system mayinclude an unmanned aerial vehicle (UAV) in the air. The UAV may includeat least one UAV camera configured to collect first ground trafficinformation, and a UAV communication module configured to transmit thecollected first ground traffic information. The example autonomousdriving system may further include a land vehicle communicativelyconnected to the UAC in the air. The land vehicle may include one ormore vehicle sensors configured to collect second ground trafficinformation surrounding the land vehicle, a land communication moduleconfigured to receive the first ground traffic information from the UAV,and a processor configured to combine the first ground trafficinformation and the second ground traffic information to generate aworld model.

To the accomplishment of the foregoing and related ends, the one or moreaspects comprise the features herein after fully described andparticularly pointed out in the claims. The following description andthe annexed drawings set forth in detail certain illustrative featuresof the one or more aspects. These features are indicative, however, ofbut a few of the various ways in which the principles of various aspectsmay be employed, and this description is intended to include all suchaspects and their equivalents.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosed aspects will hereinafter be described in conjunction withthe appended drawings, provided to illustrate and not to limit thedisclosed aspects, wherein like designations denote like elements, andin which:

FIG. 1 illustrates a diagram showing an autonomous driving system withair support in accordance with the disclosure;

FIG. 2 illustrates a diagram showing the autonomous driving system withair support in accordance with the disclosure;

FIG. 3 illustrates a diagram showing another autonomous driving systemwith air support in accordance with the disclosure;

FIG. 4 illustrates a diagram showing example components of theautonomous driving system with air support in accordance with thedisclosure;

FIG. 5 illustrates a diagram showing a conversion of traffic informationin the example autonomous driving system with air support in accordancewith the disclosure;

FIG. 6 illustrates a diagram showing a detection of accessible areas bythe example autonomous driving system with air support in accordancewith the disclosure;

FIG. 7 illustrates a diagram showing a combined detection range of theexample autonomous driving system with air support in accordance withthe disclosure;

FIG. 8 illustrates a diagram showing an example perception neuralnetwork in the example autonomous driving system with air support inaccordance with the disclosure;

FIG. 9 illustrates a diagram showing an example transformation neuralnetwork in the example autonomous driving system with air support inaccordance with the disclosure;

FIG. 10 illustrates a diagram showing another example neural network inthe example autonomous driving system with air support in accordancewith the disclosure;

FIG. 11 illustrates a diagram showing an example combined structure ofmultiple neural networks in the example autonomous driving system withair support in accordance with the disclosure;

FIG. 12 illustrates a flow chart of an example method for performingautonomous driving in the example autonomous driving system inaccordance with the disclosure; and

FIG. 13 illustrates a diagram showing the autonomous driving system withair support in an example scenario.

DETAILED DESCRIPTION

Various aspects are now described with reference to the drawings. In thefollowing description, for purpose of explanation, numerous specificdetails are set forth in order to provide a thorough understanding ofone or more aspects. It may be evident, however, that such aspect(s) maybe practiced without these specific details.

In the present disclosure, the term “comprising” and “including” as wellas their derivatives mean to contain rather than limit; the term “or,”which is also inclusive, means and/or.

In this specification, the following various embodiments used toillustrate principles of the present disclosure are only forillustrative purpose, and thus should not be understood as limiting thescope of the present disclosure by any means. The following descriptiontaken in conjunction with the accompanying drawings is to facilitate athorough understanding of the illustrative embodiments of the presentdisclosure defined by the claims and its equivalent. There are specificdetails in the following description to facilitate understanding.However, these details are only for illustrative purpose. Therefore,persons skilled in the art should understand that various alternationand modification may be made to the embodiments illustrated in thisdescription without going beyond the scope and spirit of the presentdisclosure. In addition, for clear and concise purpose, some knownfunctionality and structure are not described. Besides, identicalreference numbers refer to identical function and operation throughoutthe accompanying drawings.

FIG. 1 illustrates a diagram showing an example autonomous drivingsystem 100 with air support in accordance with the disclosure.

As depicted, the example autonomous driving system 100 includes a landvehicle 102 on the ground and an unmanned aerial vehicle (UAV) 104 inthe air. In some examples, the UAV 104 may be a drone stored in the landvehicle 102 for charging and released or launched if preferred. In someother examples, the UAV 104 may be any vehicle above the ground, e.g., asatellite.

When the UAV 104 is released and hovering above the land vehicle,multiple sensors of the UAV 104 may be configured to collect groundtraffic information. In some examples, the UAV 104 may include camerasensors, radar sensors, and/or LiDAR sensors. As the UAV 104 is in theair, the sensors of UAV 104 may capture more, or at least different,ground traffic information. For example, as depicted here, when the landvehicle 102 is traveling behind a vehicle 108, sensors on the landvehicle 102 may not be able to capture any information of a vehicle 110in front of the vehicle 108. In some other examples, an exit or a sharpturn on the road may be blocked by the vehicles 108 and 110 such thatthe sensors on the land vehicle 102 may not be able to detect the exit.Unlike the sensors on the land vehicle 102, the sensors on UAV 104 maybe able to gather ground traffic information typically unperceivable orundetectable by the sensors on the land vehicle 102.

In some examples, the sensors on the UAV 104 may be configured tocollect raw sensor information, such as visual images and/or distancesfrom ground objects to the UAV 104. The raw sensor information, e.g.,visual images and the distances, may be further converted, by aprocessor of the UAV 104, to ground traffic information in athree-dimensional (3D) coordinate system (“first ground trafficinformation” hereinafter). The position of the UAV 104 may be thecoordinate origin of the 3D coordinate system. The conversion from thecollected visual images and distance information may be performed by aprocessor of the UAV 104. Alternatively, the collected visual image andthe distance information may be transmitted to a control center 106 orthe land vehicle 102. The conversion can also be performed a processorof the control center 106 or a processor on the land vehicle 102.

In some examples, the first ground traffic information may include aposition of the land vehicle 102 in the 3D coordinate system, positionsof the still objects on the ground (e.g., curbside, lane dividing lines,stop sign, etc.), positions and motion information (such as velocity andacceleration) of the moving objects on the ground (e.g., pedestrian,other land vehicles, etc.), status information of the traffic signals(e.g., traffic light 112), and area information that indicates areasaccessible to the land vehicle 102 from the perspective of the UAV 104.The positions of the first ground traffic information may be formed assets of coordinates in the 3D coordinate system, one or more pointclouds, one or more semantic segments, or features of those objects.

While the land vehicle 102 is on the road, the sensors (e.g., camerasensors, radar sensors, and/or LiDAR sensors) on the land vehicle may beconfigured to collect raw sensor information surrounding the landvehicle 102. Similarly, for example, visual images and/or distanceinformation of surrounding objects may be collected by the sensors onthe land vehicle. The collected raw sensor information, such as visualimage and the distance information, may further converted, by aprocessor of land vehicle 102, to ground traffic information in atwo-dimensional (2D) coordinate system (“second ground trafficinformation” hereinafter). The position of the land vehicle 102 may bethe coordinate origin of the 2D coordinate system.

Similarly, the second ground traffic information may include positionsof the still objects on the ground (e.g., curbside, lane dividing lines,stop sign, etc.), positions and motion information such as velocity andacceleration of the moving objects on the ground (e.g., pedestrian,other land vehicles, etc.), status information of the traffic signals,and area information that indicates areas accessible to the land vehicle102 from the perspective of the land vehicle 102.

The ground traffic information collected by the UAV 104 and the landvehicle 102 respectively may be further combined to generate a worldmodel. The world model may include a combination of the informationcollected respectively by the UAV 104 and the land vehicle 102. In someexamples, the first ground traffic information in the 3D coordinatesystem may be converted to the 2D coordinate system with the landvehicle 102 as the coordinate origin. Such conversion may be performedby a processor of the UAV 104, a processor of the land vehicle 102, or aprocessor at the controller center 106. The process of generating theworld model is further described in more detail in accordance with FIGS.5 and 8-11 .

As the world model includes the position information of those objectsthat are difficult to be perceived by the sensors on the land vehicle102, it become more efficient, maybe safer, to control the routing, thebehavior, and the motion of the land vehicle based on the world model.For example, when the world model includes the velocities andaccelerations of the vehicles 108 and 110, the processor of the landvehicle may be configured to generate an instruction to pass the vehicle108 if the distance between the vehicles 108 and 110 is and will be safefor a time period sufficient for passing.

From the perspective of the complexity entropy system, the vehicle-UAVcooperative autonomous driving introduces the intelligent element ofentropy reduction to counter the entropy increase of the naturaliterative growth of the single-vehicle intelligent automatic drivingsystem. Through vehicle-UAV collaboration, the perception andcollaborative planning capabilities of the air-side subsystem can beintroduced to solve the problem of blind spot perception, whileexpanding the perception range and improving the safety and robustnessof decision-making and planning. In addition, vehicle-UAV collaborationis more conditional for data accumulation and collaboration, and furtherenhances individual single-vehicle intelligence and learning growthintelligence through data mining. In this way, the vehicle-UAV synergyintroduces orthogonal elements such as high-dimensional data of UAV-sideintelligence and realizes a new intelligence of entropy reductionagainst the entropy increase of system complexity.

FIG. 2 illustrates a diagram showing the example autonomous drivingsystem with air support in accordance with the disclosure.

As an example scenario depicted in FIG. 2 , when the land vehicle 102detects that the land vehicle 102 is approaching an intersection, theUAV 104 may be released to the air and fly toward the interaction beforethe land vehicle 102, typically before the land vehicle 102 reaches theintersection. In some examples, the UAV 104 may be in the air followingor leading the land vehicle 102 during the trip.

When the UAV 104 is close to or around the intersection, the sensors ofUAV 104 may be configured to collect the first ground trafficinformation including the positions of the crosswalks, the lane dividinglines, the curbsides, or a moving vehicle 202. The first ground trafficinformation may then be combined with the second ground trafficinformation by a processor of the land vehicle 102 or the control center106 to generate the world model. As the world model includes the motioninformation collected in the first ground traffic information by the UAV104, a processor of the land vehicle 102 may be configured to determinethe speed of a right turn of the land vehicle 102, or whether the landvehicle 102 needs to stop to yield in the case that the velocity of themoving vehicle 202 reaches a given threshold.

FIG. 3 illustrates a diagram showing another autonomous driving systemwith air support in accordance with the disclosure.

As depicted, one or more UAVs (e.g., UAVs 104-107) may be hovering nearthe intersection. These UAVs may be originally paired to different landvehicles respectively or a part of smart city infrastructure collectinginformation for traffic control government agencies. These UAVs may becommunicatively connected to each other, to the land vehicle 102, or tothe control center 106. In the example, the first ground trafficinformation collected/generated respectively by the UAVs 104-107 may betransmitted to the land vehicle 102, the control center 106, or any ofthe UAVs 104-107 to generate the world model.

Since first ground traffic information collected/generated respectivelyby the UAVs 104-107 theoretically includes traffic information of alarger geographic range, the world model may include information of moremoving objects. The decisions of the autonomous driving made based uponthe world model may be safer or more efficient. For example, theprocessor on the land vehicle 102 may force a hard stop if the worldmodel includes motion information of a running person around the blindspot of the land vehicle 102.

FIG. 4 illustrates a diagram showing example components of theautonomous driving system with air support in accordance with thedisclosure.

As depicted, the UAV 104 may include sensors such as one or more UAVcamera 402, one or more UAV LiDAR sensor, other UAV sensors (e.g., radarsensors). The UAV camera 402 may be configured to capture images of theground traffic. The UAV LiDAR sensor may be configured to determinedistance information of the objects on the ground, i.e., distancesbetween ground objects to the UAV 104. Other UAV sensors 406 such asradar sensors may be similarly configured to determine the distanceinformation of the ground objects. The collected images and distanceinformation may be sent to a UAV processor 410 to be converted to thefirst ground traffic information. The first ground traffic informationmay then be transmitted to the land vehicle 102 via a UAV communicationmodule 408. The UAV communication module 408 may be in communicationwith a land vehicle communication module 418 and/or a control centercommunication module 422 in accordance with wireless communicationprotocols such as Wi-Fi, Bluetooth, ZigBee, Z-Wave, MiWi, etc. In otherexamples, the images and the distance information may be sent, via theUAV communication module 408, to the land vehicle 102 or the controlcenter 106 for the conversion.

The land vehicle may include sensors such as one or more land vehiclecamera 412, one or more land vehicle LiDAR sensor, other land vehiclesensors (e.g., radar sensors). Similarly, the land vehicle camera 412may be configured to capture images of the ground traffic surroundingthe land vehicle 102. The land vehicle LiDAR sensor 414 and other landvehicle sensors 416 may be configured to distance information of thesurrounding objects. Typically, sensors on the land vehicle 102 maycollect traffic information within several hundred meters of the landvehicle 102.

Similarly, the collected images and distance information may be sent toa land vehicle processor 420 to be converted to the second groundtraffic information. In some other examples, the collected images anddistance information may be sent to the UAV 104 or the control center106 for the conversion.

Based on the first ground traffic information collected by the UAV 104and the second ground traffic information collected by the land vehicle102, the land vehicle processor 420 may be configured to generate theworld model. Notably, in at least some examples, the generating of theworld model may be performed by the UAV processor 410 or the centerprocessor 424.

Based on the world model, the land vehicle processor 420 may beconfigured to generate decisions for the land vehicle 102.

FIG. 5 illustrates a diagram showing a conversion of traffic informationin the example autonomous driving system with air support in accordancewith the disclosure.

As depicted, the processor of the UAV 104 may be configured to generatethe first ground traffic information in the 3D coordinate system. In the3D coordinate system, each of the ground objects may be associated withone or more sets of coordinates. For example, each segment of the landdividing lines may be associated with two sets of coordinates thatindicate a beginning and an end thereof. A vehicle 502 may be associatedwith four sets of coordinates that respectively indicate four corners ofa virtual boundary box that encloses the vehicle 502.

Some of the objects may be formatted as function curves in the 3Dcoordinate system, e.g.,

$\begin{bmatrix}x \\z\end{bmatrix} = {\begin{bmatrix}{{\sum}_{r = 0}^{R}a_{r}y^{r}} \\{{\sum}_{r = 0}^{R}b_{r}y^{r}}\end{bmatrix}.}$

Some other objects may be formatted as a chain of links.

For example, each segment of the lane dividing lines may be associatedwith a number of itself and a number of the next segment.

In some other examples, each of the ground objects may be represented asa semantic segment (or instance segment). The semantic segment may alsobe associated with coordinates in the 3D coordinate system.Additionally, each semantic segment may include a probability of acategory to which the object belongs. For example, a portion of thecurbside may be represented as “(x, y, z) (95%) (curbside)” showing theobject at coordinate (x, y, z) is highly likely to be a curbside.

In some other examples, each of the ground objects may be represented asa point cloud that include a set of data points in space. Each of thedata points may be associated with a set of coordinates.

Additionally, some of the ground objects may be associated with adirection to which the objects are facing. For example, the direction ofa bike, a pedestrian, or a car may be determined based on the imagescollected by the UAV camera 402.

Additionally, motion information may be associated to each moving objecton the ground. For example, velocity formatted as (v_(x), v_(y), v_(z))and acceleration formatted as (a_(x), a_(y), a_(z)) may be associatedwith the vehicle 502. In some examples, the first ground trafficinformation may include predicted trajectories of the moving objects onthe ground. The predicted trajectories of the moving objects may begenerated by the UAV processor 410 in accordance with some existingapproaches, e.g., model based approach and/or data driven approach.

Different from the first ground traffic information from the perspectiveof the UAV 104, the second ground traffic information is represented ina 2D coordinate system with the position of the land vehicle 102 beingthe coordinate origin. The second ground traffic information maysimilarly include coordinates of the ground objects, 2D function curvesin the 2D coordination system to represents some of the ground objects,semantic segments of some ground objects, point clouds of some groundobjects, directions to which some objects face, motion information ofsome moving objects on the ground, predicted trajectories of the movingobjects.

During the process of generating the world model, the first groundtraffic information may be converted to the 2D coordinate system. Thepositions of the ground objects in the 3D coordinate system may bealigned with those positions of the same objects in the 2D coordinatesystem to generate the world model.

For example, the land vehicle processor 420, the UAV processor 410, orthe center processor 424, may be configured to convert coordinates ofone or more still objects, one or more moving objects, one or moretraffic signals, and one or more accessible areas identified in the 3Dcoordinate system to coordinates in the 2D coordinate system; anddetermine coordinates of the one or more still objects, the one or moremoving objects, the one or more traffic signals, and the one or moreaccessible areas in the world model based on the converted coordinatesand the second ground traffic information.

In other examples, the land vehicle processor 420, the UAV processor410, or the center processor 424, may be configured to convert semanticsegments of one or more still objects, one or more moving objects, oneor more traffic signals, and one or more accessible areas identified inthe 3D coordinate system to semantic segment in the 2D coordinatesystem; and determine semantic segments of the one or more stillobjects, the one or more moving objects, the one or more trafficsignals, and the one or more accessible areas identified in the worldmodel based on the converted semantic segment and the second groundtraffic information.

In yet other examples, the land vehicle processor 420, the UAV processor410, or the center processor 424, may be configured to convert pointclouds of one or more still objects, one or more moving objects, one ormore traffic signals, and one or more accessible areas identified in the3D coordinate system to point clouds in the 2D coordinate system; anddetermine point clouds of the one or more still objects, the one or moremoving objects, the one or more traffic signals, and the one or moreaccessible areas identified in the world model based on the convertedpoint clouds and the second ground traffic information.

The conversion of the coordinate systems is described in greater detailbelow.

FIG. 6 illustrates a diagram showing a detection of accessible areas bythe example autonomous driving system with air support in accordancewith the disclosure.

As described above, the first ground traffic information and the secondground traffic information may include area information that indicatesthe areas accessible to the land vehicle 102. The areas may also beidentified as sets of coordinates. Thus, the world model may alsoindicate the accessible areas as marked in patterns in FIG. 6 . In someexamples, the determination of the accessible areas 604 may be based ontraffic rules and motion information of the ground objects anddynamically adjusted. For example, when the vehicle 602 is detected toapply a hard brake, the accessible areas 604 may be adjusted such thatthe land vehicle 102 may keep a safe distance.

FIG. 7 illustrates a diagram showing a combined detection range of theexample autonomous driving system with air support in accordance withthe disclosure.

As shown, due the limits of the sensors on the land vehicle 102, thedetection range 702 of the land vehicle 102 may be within severalhundred meters from the land vehicle 102. Since the sensors on the UAV104 may collect information that normally cannot be perceived by thesensors on the land vehicle 102, a detection range 704 of the UAV 104may be much larger than the detection rage 702.

Further, because the world model essentially includes a combination ofthe first ground traffic information and the second ground trafficinformation, the range of the world model may be greater than, or atleast equal to, the detection range 704 of the UAV 104.

FIG. 8 illustrates a diagram showing an example perception neuralnetwork in the example autonomous driving system with air support inaccordance with the disclosure.

As shown, the images and the position information (e.g., images 810 and812, LiDAR points such as point clouds 814 and 816) respectivelycollected by the sensors of the UAV 102 and the land vehicle 102 (e.g.,UAV camera 402, UAV LiDAR sensor 404, UAV sensors 406, land vehiclecamera 412, land vehicle LiDAR sensor 414, land vehicle sensors 416,etc.) may be input to a perception neural network 802 via one or morefeature extraction networks 818. The feature extraction networks 818 maybe configured to extract features from the images and the positioninformation. The extracted features may then be input into theperception neural network 802.

A system administrator (a person) may label the objects on the images810 and 812 based on his/her experience to set ground truth values ofthe perception neural network 802. With sufficient teaching input by thesystem administrator, the perception neural network 802 may detect theobjects described in the images and the position information and outputperceived objects 806 (e.g., other vehicles on the road, accessibleareas, lane dividing lines, etc.) as the results.

FIG. 9 illustrates a diagram showing an example transformation neuralnetwork in the example autonomous driving system with air support inaccordance with the disclosure.

As described above, the first ground traffic information may beconverted into the 2D coordinate system to be consistent with the secondground traffic information; the first ground traffic information and thesecond ground traffic information may then be combined to generate theworld model. The world model may include information of the objectsperceivable to the sensors of the UAV 104 or to the sensors of the landvehicle 102. A transformation network 916 may be configured to output atransformation matrix 918 that can convert the coordinates in the 3Dcoordinate system to coordinates in the 2D coordinate system.

Images 908 collected by the UAV camera, e.g., 402, may be sent to afeature extraction network 912 to extract features of the objectscontained in the images 908. When properly trained, the featureextraction network 912 may output features of those perceived objects byUAV 904.

Similarly, images 910 collected by the land vehicle camera, e.g., 412,may be sent to a feature extraction network 914 to extract features ofthe objects contained in the images 910. When properly trained, thefeature extraction network 914 may output features of the perceivedobjects by land vehicle 902.

Features of both the perceived objects by UAV 904 and the perceivedobjects by land vehicle 902 may be combined and input to thetransformation network 916. After training, the transformation network916 may output the transformation matrix 918. With the transformationmatrix 918, features of the perceived objects by UAV 904 may beconverted to features of objects in the 2D coordinate system. Thefeatures of the perceived objects by UAV in 2D system 920 may becompared to the perceived objects by land vehicle 902 to determine ifthe features of the objects perceived by the UAV 104 and the landvehicle 102 are consistent after converting the coordinates. The resultsof the comparison may be fed back to the transformation network 916 asconstraints to further train the transformation network 916 to yield abetter transformation matrix 918.

Notably, in at least some examples, the transformation network 916 maybe configured to generate a transformation matrix intended to convertthe coordinates in the 2D system to the 3D system. Processes andoperations are similar to those described above.

FIG. 10 illustrates a diagram showing another example neural network inthe example autonomous driving system with air support in accordancewith the disclosure.

As described above, the first ground traffic information may beconverted into the 2D coordinate system to be consistent with the secondground traffic information; the first ground traffic information and thesecond ground traffic information may then be combined to generate theworld model. The world model may include information of the objectsperceivable to the sensors of the UAV 104 and/or to the sensors of theland vehicle 102.

Alternatively, images 1002 collected by the UAV camera, e.g., 402, andimages 1004 collected by the land vehicle camera, e.g., 412, may besubmitted to a fusion neural network directly without a transformationof coordinates.

A system administrator (a person) may label the objects on the images1002 and 1004 based on his/her experience to set ground truth values ofthe fusion neural network 1006. With sufficient teaching input by thesystem administrator, the fusion neural network 1006 may eventuallydetect the objects described in the images and the position informationand output perceived objects 1008 (e.g., other vehicles on the road,accessible areas, lane dividing lines, etc.) as the results.

FIG. 11 illustrates a diagram showing an example combined structure ofmultiple neural networks in the example autonomous driving system withair support in accordance with the disclosure.

As depicted, the structure described in accordance with FIG. 9 may becombined with the fusion neural network described in FIG. 10 .

Images 1108 collected by the UAV 104 and images 1110 collected by theland vehicle 102 may be input to feature extraction networks 1112 and1114 respectively. Features of the perceived objects by UAV 1104 may begenerated by the feature extraction network 1112; features of theperceived objects by land vehicle 1102 may be generated by the featureextraction network 1114. As described above in accordance with FIG. 9 ,the features can be utilized to generate a transformation matrix 1118.Meanwhile, the features can also be input to a fusion neural network1126 to recognize objects in the images 1108 and 1110.

Although the transformation matrix 1118 may be not required to recognizethe objects in the images 1108 and 1110, the transformation matrix 1118may be utilized for future route planning and other decisions forautonomous driving.

FIG. 12 illustrates a flow chart of an example method for performingautonomous driving in the example autonomous driving system inaccordance with the disclosure. The flowchart illustrates a process ofimplementing autonomous driving with air support.

At block 1202, the operations of the example method may includecollecting, by at least one UAV camera, first raw sensor information.For example, the sensors on the UAV 104 may be configured to collectvisual images and/or distances from ground objects to the UAV 104. Thevisual images and the distances may be further converted to first groundtraffic information in a 3D coordinate system.

At block 1204, the operations of the example method may includetransmitting, by a UAV communication module, the collected first groundtraffic information to a land vehicle. For example, the first groundtraffic information may then be transmitted to the land vehicle 102 viathe UAV communication module 408.

At block 1206, the operations of the example method may includecollecting, by one or more vehicle sensors, second raw sensorinformation surrounding the land vehicle. For example, the land vehiclecamera 412 may be configured to capture images of the ground trafficsurrounding the land vehicle 102. The land vehicle LiDAR sensor 414 andother land vehicle sensors 416 may be configured to determine distanceinformation of the surrounding objects. Similarly, the collected imagesand distance information may be sent to a land vehicle processor 420 tobe converted to the second ground traffic information.

At block 1208, the operations of the example method may includereceiving, by a land communication module, the first ground trafficinformation from the UAV. For example, the land vehicle communicationmodule 418 may be configured to receive the first ground trafficinformation from the UAV 104 via one or more wireless communicationlinks.

At block 1210, the operations of the example method may include fusing,by a processor, the first ground traffic information and the secondground traffic information to generate a world model. For example, theland vehicle processor 420 may be configured to generate the worldmodel. The world model may include a combination of the informationcollected respectively by the UAV 104 and the land vehicle 102. In someexamples, the first ground traffic information in the 3D coordinatesystem may be converted to the 2D coordinate system with the landvehicle 102 as the coordinate origin. Such conversion may be performedby a processor of the UAV 104, a processor of the land vehicle 102, or aprocessor at the controller center 106.

FIG. 13 illustrates a diagram showing the autonomous driving system withair support in an example scenario. As depicted, in some examplescenario, while multiple vehicles are travelling around the land vehicle102, sensors on the land vehicle 102 may be blocked and cannot identifynecessary information to determine the position of the land vehicle 102itself, which may further lead to incorrect driving decisions such asrunning a red light or missing an exit.

In this example scenario, since the first ground traffic informationincludes the position of the land vehicle and other objectsunperceivable by the sensors on land vehicle 102, the world model mayinclude the information necessary for generating correct drivingdecisions, e.g., changing lane when approaching the exist.

The process and method as depicted in the foregoing drawings may beexecuted through processing logics including hardware (e.g., circuit,special logic, etc.), firmware, software (e.g., a software embodied in anon-transient computer readable medium), or combination of each two.Although the above describes the process or method in light of certainsequential operation, it should be understood that certain operationdescribed herein may be executed in different orders. Additionally, someoperations may be executed concurrently rather than sequentially.

In the above description, each embodiment of the present disclosure isillustrated with reference to certain illustrative embodiments. Any ofthe above-mentioned components or devices may be implemented by ahardware circuit (e.g., application specific integrated circuit (ASIC)).Apparently, various modifications may be made to each embodiment withoutgoing beyond the wider spirit and scope of the present disclosurepresented by the affiliated claims. Correspondingly, the description andaccompanying figures should be understood as illustration only ratherthan limitation. It is understood that the specific order or hierarchyof steps in the processes disclosed is an illustration of exemplaryapproaches. Based upon design preferences, it is understood that thespecific order or hierarchy of steps in the processes may be rearranged.Further, some steps may be combined or omitted. The accompanying methodclaims present elements of the various steps in a sample order and arenot meant to be limited to the specific order or hierarchy presented.

The previous description is provided to enable any person skilled in theart to practice the various aspects described herein. Variousmodifications to these aspects will be readily apparent to those skilledin the art, and the generic principles defined herein may be applied toother aspects. Thus, the claims are not intended to be limited to theaspects shown herein but is to be accorded the full scope consistentwith the language claims, wherein reference to an element in thesingular is not intended to mean “one and only one” unless specificallyso stated, but rather “one or more.” Unless specifically statedotherwise, the term “some” refers to one or more. All structural andfunctional equivalents to the elements of the various aspects describedherein that are known or later come to be known to those of ordinaryskill in the art are expressly incorporated herein by reference and areintended to be encompassed by the claims. Moreover, nothing disclosedherein is intended to be dedicated to the public regardless of whethersuch disclosure is explicitly recited in the claims. No claim element isto be construed as a means plus function unless the element is expresslyrecited using the phrase “means for.”

Moreover, the term “or” is intended to mean an inclusive “or” ratherthan an exclusive “or.” That is, unless specified otherwise, or clearfrom the context, the phrase “X employs A or B” is intended to mean anyof the natural inclusive permutations. That is, the phrase “X employs Aor B” is satisfied by any of the following instances: X employs A; Xemploys B; or X employs both A and B. In addition, the articles “a” and“an” as used in this application and the appended claims shouldgenerally be construed to mean “one or more” unless specified otherwiseor clear from the context to be directed to a singular form.

We claim:
 1. An autonomous driving system, comprising: an unmannedaerial vehicle (UAV) in the air, wherein the UAV includes: at least oneUAV camera configured to collect first raw sensor information, a UAVprocessor configured to convert the first raw sensor information tofirst ground traffic information, and a UAV communication moduleconfigured to transmit the collected first ground traffic information;and a land vehicle communicatively connected to the UAC in the air,wherein the land vehicle includes: one or more vehicle sensorsconfigured to collect second raw sensor information surrounding the landvehicle, a land vehicle processor configured to convert the second rawsensor information to second ground traffic information, and a landcommunication module configured to receive the first ground trafficinformation from the UAV, wherein the land vehicle a processor isfurther configured to combine the first ground traffic information andthe second ground traffic information to generate a world model.
 2. Theautonomous driving system of claim 1, wherein the first ground trafficinformation is formatted in a coordinate system with a position of theUAV as a coordinate origin.
 3. The autonomous driving system of claim 2,wherein the first ground traffic information includes at least one of: aposition of the land vehicle in the coordinate system; first positioninformation that indicates locations of one or more still objects on theground in the coordinate system with the position of the UAV as thecoordinate origin; first motion information that indicates velocities ofone or more moving objects on the ground; first predicted trajectoriesof the one or more moving objects on the ground; first statusinformation that indicates statuses of one or more traffic signals; andfirst area information that indicates one or more accessible areas tothe land vehicle from a perspective of the UAV.
 4. The autonomousdriving system of claim 3, wherein the first position information of theone or more still objects is formed as one or more sets of coordinatesin the coordinate system, one or more point clouds, one or more semanticsegments, or features of the still objects extracted by a neuralnetwork.
 5. The autonomous driving system of claim 1, wherein the secondground traffic information is formatted in a coordinate system with aposition of the land vehicle as a coordinate origin.
 6. The autonomousdriving system of claim 5, wherein the second ground traffic informationinclude at least one of: second position information that indicateslocations of one or more still objects surrounding the land vehicle inthe coordinate system with the position of the land vehicle as thecoordinate origin; second motion information that indicates velocitiesof one or more moving objects surrounding the land vehicle; secondpredicted trajectories of the one or more moving objects surrounding theland vehicle; second status information that indicates statuses of oneor more traffic signals; and second area information that indicates oneor more accessible areas to the land vehicle from a perspective of theland vehicle.
 7. The autonomous driving system of claim 6, wherein thesecond position information is formed as one or more sets of coordinatesin the coordinate system, one or more point clouds, one or more semanticsegments, or features of the still objects extracted by a neuralnetwork.
 8. The autonomous driving system of claim 1, wherein the landvehicle processor is configured to: convert the first ground trafficinformation from a first coordinate system with a position of the UAV asa coordinate origin to a second coordinate system with a position of theland vehicle as the coordinate origin, convert coordinates of one ormore still objects, one or more moving objects, one or more trafficsignals, and one or more accessible areas and predicted trajectories ofthe one or more moving objects identified in the first coordinate systemto coordinates in the second coordinate system; and determinecoordinates of the one or more still objects, the one or more movingobjects, the one or more traffic signals, and the one or more accessibleareas in the world model based on the converted coordinates and thesecond ground traffic information.
 9. The autonomous driving system ofclaim 1, wherein the land vehicle processor is configured to: convertsemantic segments of one or more still objects, one or more movingobjects, one or more traffic signals, and one or more accessible areasidentified in the first coordinate system to semantic segment in thesecond coordinate system; and determine semantic segments of the one ormore still objects, the one or more moving objects, the one or moretraffic signals, and the one or more accessible areas and predictedtrajectories of the one or more moving objects identified in the worldmodel based on the converted semantic segment and the second groundtraffic information.
 10. The autonomous driving system of claim 1,wherein the land vehicle processor is configured to: convert pointclouds of one or more still objects, one or more moving objects, one ormore traffic signals, and one or more accessible areas identified in thefirst coordinate system to point clouds in the second coordinate system;and determine point clouds of the one or more still objects, the one ormore moving objects, the one or more traffic signals, and the one ormore accessible areas and predicted trajectories of the one or moremoving objects identified in the world model based on the convertedpoint clouds and the second ground traffic information.
 11. A landvehicle in an autonomous driving system, comprising: a landcommunication module configured to receive first ground trafficinformation surrounding the land vehicle, wherein the first groundtraffic information is generated by an unmanned aerial vehicle (UAV) inthe air; one or more vehicle sensors configured to collect surroundingthe land vehicle; a processor configured to combine the first groundtraffic information and the second ground traffic information togenerate a world model.
 12. The land vehicle of claim 11, wherein thefirst ground traffic information is formatted in a coordinate systemwith a position of the UAV as a coordinate origin.
 13. The land vehicleof claim 12, wherein the first ground traffic information includes atleast one of: a position of the land vehicle in the coordinate system;first position information that indicates locations of one or more stillobjects on the ground in the coordinate system with the position of theUAV as the coordinate origin; first motion information that indicatesvelocities of one or more moving objects on the ground; first predictedtrajectories of the one or more moving objects on the ground; firststatus information that indicates statuses of one or more trafficsignals; and first area information that indicates one or moreaccessible areas to the land vehicle from a perspective of the UAV. 14.The land vehicle of claim 13, wherein the first position information ofthe one or more still objects is formed as one or more sets ofcoordinates in the coordinate system, one or more point clouds, one ormore semantic segments, or features of the still objects extracted by aneural network.
 15. The land vehicle of claim 11, wherein the secondground traffic information is formatted in a coordinate system with aposition of the land vehicle as a coordinate origin.
 16. The landvehicle of claim 15, wherein the second ground traffic informationinclude at least one of: second position information that indicateslocations of one or more still objects surrounding the land vehicle inthe coordinate system with the position of the land vehicle as thecoordinate origin; second motion information that indicates velocitiesof one or more moving objects surrounding the land vehicle; secondpredicted trajectories of the one or more moving objects surrounding theland vehicle; second status information that indicates statuses of oneor more traffic signals; and second area information that indicates oneor more accessible areas to the land vehicle from a perspective of theland vehicle.
 17. The land vehicle of claim 16, wherein the secondposition information is formed as one or more sets of coordinates in thecoordinate system, one or more point clouds, one or more semanticsegments, or features of the still objects extracted by a neuralnetwork.
 18. The land vehicle of claim 11, wherein the processor isconfigured to: convert the first ground traffic information from a firstcoordinate system with a position of the UAV as a coordinate origin to asecond coordinate system with a position of the land vehicle as thecoordinate origin, convert coordinates of one or more still objects, oneor more moving objects, one or more traffic signals, and one or moreaccessible areas and predicted trajectories of the one or more movingobjects identified in the first coordinate system to coordinates in thesecond coordinate system; and determine coordinates of the one or morestill objects, the one or more moving objects, the one or more trafficsignals, and the one or more accessible areas in the world model basedon the converted coordinates and the second ground traffic information.19. The land vehicle of claim 11, wherein the processor is configuredto: convert semantic segments of one or more still objects, one or moremoving objects, one or more traffic signals, and one or more accessibleareas identified in the first coordinate system to semantic segment inthe second coordinate system; and determine semantic segments of the oneor more still objects, the one or more moving objects, the one or moretraffic signals, and the one or more accessible areas and predictedtrajectories of the one or more moving objects identified in the worldmodel based on the converted semantic segment and the second groundtraffic information.
 20. The land vehicle of claim 11, wherein theprocessor is configured to: convert point clouds of one or more stillobjects, one or more moving objects, one or more traffic signals, andone or more accessible areas identified in the first coordinate systemto point clouds in the second coordinate system; and determine pointclouds of the one or more still objects, the one or more moving objects,the one or more traffic signals, and the one or more accessible areasand predicted trajectories of the one or more moving objects identifiedin the world model based on the converted point clouds and the secondground traffic information.